A Framework for Evaluating Vendor Procurement in a Digital Health Project.

The eHealth Centre of Excellence, a Waterloo, Ontario-based organization that advances and promotes digital health initiatives in clinical care, developed and assessed an innovative evaluation procurement framework. The purpose of the framework was to assess and support long-term vendor-organization procurement partnerships to develop, improve and expand electronic referral (eReferral) solutions. The framework focused on six criteria: the quality of the eReferral solution, its implementation, the service provided, the extent of training and knowledge transfer, the quality of the vendor's team and the vendor's project experience. These domains were further defined by components and key performance indicators unique to the eReferral solution to accommodate the stakeholders' specified needs as well as change management challenges to create value for users and organizations in long-term relationships. The evaluation used both qualitative and quantitative methodologies. The framework used data from three sources: (1) the System Coordinated Access program and vendor team experience surveys that focused on the six criteria mentioned earlier; (2) key stakeholder interviews that focused on system quality, user satisfaction and perception of net benefits; and (3) a vendor scorecard that focused on deliverables and efficiencies. Vendor procurement should be viewed not as a process that ends when a vendor is selected but rather as a continuing and evolving relationship. Evaluation should assess the ability and willingness of vendors to support stakeholders and meet their needs, stimulate new ideas and adapt to changing environments and expanding systems. The model enabled recording of factors necessary for successful outcomes and provided a strategy to help select vendors for successful long-term partnerships.

Assessing Patient Satisfaction and Experience With an Electronic Referral Process.

BACKGROUND AND OBJECTIVES: Our study aimed to identify patients' perception of an eReferral process and e-mail notification system. METHODS: Patients within the Waterloo Wellington Local Health Integration Network who registered their e-mail address with physicians who adopted the eReferral system, and therefore received e-mail notifications of their booked appointment, were invited to complete an online satisfaction survey. This patient experience survey is an ongoing online link embedded within the confirmation e-mail of the booked appointment. The survey is hosted on the eReferral solution platform and has been operational since November 2017. The survey consists of 8 questions with 3 main categories to assess patients' opinion of their experience of the referral process and notification system using a 5-point Likert scale and open-ended questions. RESULTS: A total of 545 patients have completed the patient satisfaction survey within this reporting period with a response rate of 15%. In general, 94% of patients agreed that receiving a confirmation e-mail of their booked appointment had improved their experience with the referral process. The majority (94%) agreed that the eReferral process was easy to follow, and 83% agreed that they were able to get the care they needed within a reasonable time. Compared with their past referral experiences, 80% of patients felt more informed throughout this electronic referral process. Using binominal logistic regression, participants whose preferences were considered had 8.06 times higher odds to exhibit satisfaction with the referral process than those who did not. Patients' qualitative responses identified the eReferral process as being quick, efficient, and resulting in a sense of being in control of their own health care. There are some limitations to the system felt by some of the patients who responded to the open-ended questions of the survey. Patients identified the need to add a complementary structure to the notification design consisting of multiple dates and times with a chance to pick the appointment that suits patients best instead of being restricted to only 1 appointment date. A few patients thought that the heading of the e-mail notification system should be more distinguishable for easier tracking. Furthermore, some patients felt the need to add some notes to the initial e-mail advising patients of the next steps throughout their referral process. CONCLUSION: eReferral has improved patients' experience with the referral process. Our findings in this study would support the solution vendor in its efforts to refine and enhance active communication channels with patients for sustainable health care that meets patients' expectations and needs.

Electronic referral systems in health care: a scoping review.

Introduction: Electronic referral (eReferral) systems have been designed with the dual purpose of decreasing wait times and improving workflow efficiency. Evidence about the clinical and economic value enabled through the use of eReferral systems is limited. Our objective was to review the evidence base for the effectiveness and cost-effectiveness of eReferral systems. This review is part of a bigger project to inform the economic benefits of a regional eReferral implementation program. Material and methods: A systematic search was conducted to capture the available literature on the effectiveness and cost-effectiveness of eReferral system interventions. Evaluation of eReferral system for cost or outcome(s) were included. Strictly e-consultation systems were excluded. We only included publications in English. Results: We found 274 citations. After removing duplicates and conducting levels one and two screenings, nine publications qualified. Results were divided into four categories: cost or cost-effectiveness analysis, changes in workflow efficiency, the quantity of referrals, and the quality of referrals. A full economic evaluation, conducted in Denmark, found that an eReferral system was cost-effective compared with a paper-based referral system. Of the other eight studies, three demonstrated positive changes in referral processing; two evaluated changes in the quality of the referrals, and three evaluated if the eReferral system increased the quantity of referrals. Discussion: The evidence base on the effectiveness of eReferral systems to improve communication between primary care and specialists and to decrease wait times is positive but limited. Economic evaluations are needed to examine the clinical and economic value of eReferral systems in health care.

Using Digital Health to Support Best Practices: Impact of MRI Ordering Guidelines Embedded Within an Electronic Referral Solution.

BACKGROUND: Between 2003 and 2012, the number of MRIs performed in Canada more than doubled to 1.7 million [1]. According to a 2010 Health Council of Canada report nearly 30% of MRIs were inappropriately ordered [2]. The use of diagnostic imaging referral guidelines has been shown to improve the appropriateness of imaging orders [3, 4]. OBJECTIVES: To identify the number of unnecessary pre-consult MRIs ordered for patients with knee pain. As well, the impact that new evidence-based clinical decision support (DS) guidelines embedded within the referral form has had on the number of unnecessary MRIs was investigated. METHODS: This study employed a retrospective design approach. Charts of all knee pain patients over the age of 55 who were referred for consultation to the 5 participating orthopedic surgeons during the study period were reviewed by three medical students. RESULTS: 270 patient charts were included in this study. MRI was ordered for 60 patients with only 56.7% having had a prior X-ray. Of the 60 ordered MRIs, 50 (84%) were considered inappropriate, while only 10 (16%) were appropriate. Our results were compared to previous results of a quality improvement study implemented at the same clinic. A substantial reduction of 12% in the number of pre-consult MRIs and a 5% increase in the number of ordered X-rays before consultation was demonstrated. CONCLUSION: This work highlights the impact of including DS tools within an electronic referral form to support clinical best practices.

The Connecting South West Ontario (cSWO) Benefits Model: An Approach for the Collaborative Capture of Value of Electronic Health Records and Enabling Technology.

This paper explains the benefits model developed and deployed by the connecting South West Ontario (cSWO) program. The cSWO approach is founded on the principles of enabling clinical and organizational value and the recognition that enabling requires a collaborative approach that can include several perspectives. We describe our approach which is aimed at creating a four-part harmony between change management and adoption, best practice research and quality indicators, data analytics and clinical value production.

Beyond "Two Cultures": Guidance for Establishing Effective Researcher/Health System Partnerships.

BACKGROUND: The current literature proposing criteria and guidelines for collaborative health system research often fails to differentiate between: (a) various types of partnerships, (b) collaborations formed for the specific purpose of developing a research proposal and those based on long-standing relationships, (c) researcher vs. decision-maker initiatives, and (d) the underlying drivers for the collaboration. METHODS: Qualitative interviews were conducted with 16 decision-makers and researchers who partnered on a Canadian major peer-reviewed grant proposal in 2013. Objectives of this exploration of participants' experiences with health system research collaboration were to: (a) explore perspectives and experience with research collaboration in general; (b) identify characteristics and strategies associated with effective partnerships; and (c) provide guidance for development of effective research partnerships. Interviews were audio-recorded and transcribed: transcripts were qualitatively analyzed using a general inductive approach. RESULTS: Findings suggest that the common "two cultures" approach to research/decision-maker collaboration provides an inadequate framework for understanding the complexity of research partnerships. Many commonly-identified challenges to researcher/knowledge user (KU) collaboration are experienced as manageable by experienced research teams. Additional challenges (past experience with research and researchers; issues arising from previous collaboration; and health system dynamics) may be experienced in partnerships based on existing collaborations, and interact with partnership demands of time and communication. Current research practice may discourage KUs from engaging in collaborative research, in spite of strong beliefs in its potential benefits. Practical suggestions for supporting collaborations designed to respond to real-time health system challenges were identified. CONCLUSION: Participants' experience with previous research activities, factors related to the established collaboration, and interpersonal, intra- and inter-organizational dynamics may present additional challenges to research partnerships built on existing collaboration. Differences between researchers and KUs may pose no greater challenges than differences among KUs (at various levels, and representing diverse perspectives and organizations) themselves. Effective "relationship brokering" is essential for meaningful collaboration.